The Rise of Unmoderated Usability Testing

Unmoderated usability testing is becoming more popular. It lets real users test products on their own devices, in their own space, without a facilitator. This helps teams get faster and more cost-effective insights.

The benefits include:

• More participants in less time
• Users in their natural digital environment
• Insights from a global audience, overcoming time zone challenges

But removing a moderator comes with trade-offs. A live moderator offers a human touch, adaptability, and the ability to clarify issues in real time.

How can AI help bridge this gap?

Can AI Improve Usability Testing?

AI, particularly generative AI models like GPT-4, can mimic human conversation. If used correctly, AI could:

• Ask follow-up questions to get richer feedback
• Clarify vague responses from participants
• Improve the depth of qualitative user insights

But does it actually work?

We tested GPT-4 in real-time usability research to analyze its strengths and limitations.

Want to optimize your usability testing process? Contact our team to explore how AI-driven testing can transform your research workflow.

Our Study: Testing GPT-4’s Follow-Up Questions

We ran usability tests on an e-commerce prototype, testing three different approaches:

1. A static questionnaire (pre-set questions only)
2. AI-generated follow-up questions based on the first open-ended response
3. AI follow-ups added to all pre-defined questions

How did AI perform?

While the AI-driven questions sometimes led to richer responses, they also had major challenges:

• Limited ability to uncover new usability issues
• Repetitive and frustrating questions for participants
• Lower perceived question quality

One big takeaway? AI follow-ups often ignored context, leading to redundant or irrelevant questions.

Challenges in AI-Powered Usability Testing

AI Often Lacks Context Awareness

Context is everything in usability testing. AI-generated questions must consider:

• General testing principles: Avoid leading questions or unrealistic hypotheticals.
• Research goals: AI should focus on what truly matters in a test.
• Tasks and interactions: Questions should mirror what users actually did.
• Previously answered responses: Avoid asking users to repeat themselves.

AI Can Frustrate Participants

Users became annoyed by repetitive follow-ups. Many answered with phrases like, “I already said that!” Frustration can skew feedback, reducing the reliability of insights.

How to Make AI Usability Testing Work

AI can be useful if applied carefully. Here’s how to make it work for you:

• Use a mix of static questions and AI-driven follow-ups to balance precision with adaptability.
• Ensure AI questions are context-aware to avoid irrelevant or repetitive follow-ups.
• Allow researchers manual control so AI doesn’t replace human oversight.

Want to test AI-powered usability testing for your product?

Explore our usability research services to enhance your product’s UX.

The Future of AI in UX Research

AI isn’t replacing human researchers—at least not yet. AI can assist, but humans need to stay in control to ensure quality insights.

The sweet spot? Humans + AI = Better Insights.

AI is just one tool in your UX toolbox. The key is using it in the right way.

Let’s build smarter, more efficient usability tests together. Request a quote today.

For more in-depth information, please read this in-depth research article.